{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AI Platform Prediction Load Testing using Locust\n",
    "\n",
    "This Notebook demonstrates how to perform load testing of AI Platform Prediction using [Locust](https://locust.io). The Notebook covers the following steps:\n",
    "1. Creating custom log-based metrics in Cloud Monitoring for the Locust logs\n",
    "2. Creating a Cloud Monitoring Dashboard to display the AI Platform and custom metrics\n",
    "3. Deploying Locust to GKE clustor\n",
    "4. Configuring Lucost test\n",
    "5. Runing the Lucost load test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "This notebook was tested on **AI Platform Notebooks** using the standard TF 2.8 image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install required packages\n",
    "\n",
    "You can safely ignore the dependency errors. Confirm the last message starting from \"Successfully installed...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
     "!pip install --user locust==2.11.1\\\n",
     "  google-cloud-monitoring==2.11.1\\\n",
     "  google-cloud-logging==3.2.2\\\n",
     "  google-cloud-monitoring-dashboards==2.7.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import IPython\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(True)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "import time\n",
    "import datetime\n",
    "import json\n",
    "import requests\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import google.auth\n",
    "\n",
    "from typing import List\n",
    "\n",
    "from google.api_core.exceptions import GoogleAPICallError \n",
    "\n",
    "from google.cloud import logging_v2\n",
    "from google.cloud.logging_v2.services.metrics_service_v2 import MetricsServiceV2Client\n",
    "from google.cloud.logging_v2.services.logging_service_v2 import LoggingServiceV2Client\n",
    "\n",
    "from google.cloud.monitoring_dashboard_v1.types import Dashboard\n",
    "from google.cloud.monitoring_dashboard_v1 import DashboardsServiceClient\n",
    "from google.cloud.monitoring_v3 import MetricServiceClient\n",
    "from google.cloud.monitoring_v3.query import Query\n",
    "from google.cloud.monitoring_v3.types import TimeInterval\n",
    "\n",
    "from google.protobuf.json_format import ParseDict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure GCP environment settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = '[your-google-project-id]' # Set your project Id\n",
    "BUCKET = '[your-bucket-name]' # Set your bucket name Id\n",
    "REGION = '[your-region]'  # Set your region for deploying the model\n",
    "GKE_CLUSTER_NAME = '[your-gke-cluster-name]' # Set the GKE cluster name\n",
    "GKE_CLUSTER_ZONE = '[your-gke-cluster-zone]' # Set the GKE cluster zone\n",
    "MODEL_NAME = 'resnet_classifier'\n",
    "MODEL_VERSION = 'v1'\n",
    "IMAGES_FOLDER = 'test_images'\n",
    "GCS_LOCUST_TEST_CONFIG_DIR = 'gs://{}/locust-test'.format(BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Creating Custom Log-based Metrics in Cloud Monitoring\n",
    "\n",
    "In this section of the notebook you will use the [Python Cloud Logging client library](https://googleapis.dev/python/logging/latest/v2.html) to create a set of custom log-based metrics. The metrics are based on the log entries generated by the example locustfile script. The script writes the log entries into the *Cloud Logging* log named `locust`.\n",
    "\n",
    "Each log entry includes a set of key value pairs encoded as the JSON payload type. The metrics are based on the subset of keys from the log entry.\n",
    "\n",
    "Key | Value\n",
    "----|------\n",
    "test_id | An ID of a test\n",
    "model | An AI Platform Prediction Model name\n",
    "model_version | An AI Platform Prediction Model version\n",
    "latency | A 95 percentile response time, which is calculated over a 10 sliding second window\n",
    "num_requests | A total number of requests since the test started\n",
    "num_failures | A total number of requests since the test started\n",
    "user_count | A number of simulated users \n",
    "rps | A current requests per second\n",
    "\n",
    "\n",
    "Refer to the [Cloud Logging API reference](https://googleapis.dev/python/logging/latest/v2.html) for more information about the API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Define a helper function that creates a custom log metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_locust_metric(\n",
    "    metric_name:str,\n",
    "    log_path:str, \n",
    "    value_field:str,  \n",
    "    bucket_bounds:List[int]):\n",
    "    \n",
    "    metric_path = logging_client.log_metric_path(PROJECT_ID, metric_name)\n",
    "    log_entry_filter = 'resource.type=global AND logName={}'.format(log_path)\n",
    "    \n",
    "    metric_descriptor = {\n",
    "        'metric_kind': 'DELTA',\n",
    "        'value_type': 'DISTRIBUTION',\n",
    "        'labels': [\n",
    "            {\n",
    "                'key': 'test_id',\n",
    "                'value_type': 'STRING'\n",
    "            },\n",
    "            {\n",
    "                'key': 'signature',\n",
    "                'value_type': 'STRING'\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "    \n",
    "    bucket_options = {\n",
    "        'explicit_buckets': {\n",
    "            'bounds': bucket_bounds\n",
    "        }\n",
    "    }\n",
    "    \n",
    "    value_extractor = 'EXTRACT(jsonPayload.{})'.format(value_field)\n",
    "    label_extractors = {\n",
    "        'test_id': 'EXTRACT(jsonPayload.test_id)',\n",
    "        'signature': 'EXTRACT(jsonPayload.signature)'\n",
    "    }\n",
    "    \n",
    "    metric = logging_v2.types.LogMetric(\n",
    "        name=metric_name,\n",
    "        filter=log_entry_filter,\n",
    "        value_extractor=value_extractor,\n",
    "        bucket_options=bucket_options,\n",
    "        label_extractors=label_extractors,\n",
    "        metric_descriptor=metric_descriptor,\n",
    "    )\n",
    "    \n",
    "    try:\n",
    "        logging_client.get_log_metric(metric_path)\n",
    "        print('Metric: {} already exists'.format(metric_path))\n",
    "    except:\n",
    "        request = logging_v2.types.logging_metrics.CreateLogMetricRequest(\n",
    "            parent=parent,\n",
    "            metric=metric,\n",
    "        )\n",
    "        logging_client.create_log_metric(request)\n",
    "        print('Created metric {}'.format(metric_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Create a logging client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_name = 'locust'\n",
    "\n",
    "creds , _ = google.auth.default()\n",
    "logging_client = MetricsServiceV2Client(credentials=creds)\n",
    "\n",
    "parent = logging_client.common_project_path(PROJECT_ID)\n",
    "log_path = LoggingServiceV2Client.log_path(PROJECT_ID, log_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Create metrics to track Locust logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user count metric\n",
    "metric_name = 'locust_users'\n",
    "value_field = 'user_count'\n",
    "bucket_bounds = [1, 16, 32, 64, 128]\n",
    "\n",
    "create_locust_metric(metric_name, log_path, value_field, bucket_bounds)\n",
    "\n",
    "# latency metric\n",
    "metric_name = 'locust_latency'\n",
    "value_field = 'latency'\n",
    "bucket_bounds = [1, 50, 100, 200, 500]\n",
    "\n",
    "create_locust_metric(metric_name, log_path, value_field, bucket_bounds)\n",
    "\n",
    "# failure count metric\n",
    "metric_name = 'num_failures'\n",
    "value_field = 'num_failures'\n",
    "bucket_bounds = [1, 1000]\n",
    "\n",
    "create_locust_metric(metric_name, log_path, value_field, bucket_bounds)\n",
    "\n",
    "# request count metric\n",
    "metric_name = 'num_requests'\n",
    "value_field = 'num_requests'\n",
    "bucket_bounds = [1, 1000]\n",
    "\n",
    "create_locust_metric(metric_name, log_path, value_field, bucket_bounds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4. List metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request = {'parent': parent}\n",
    "metrics = logging_client.list_log_metrics(request)\n",
    "\n",
    "if not list(metrics):\n",
    "    print(\"There are not any log based metrics defined in the the project\")\n",
    "else:\n",
    "    for element in logging_client.list_log_metrics(request):\n",
    "        print(element.metric_descriptor.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Creating the Cloud Monitoring dashboard\n",
    "\n",
    "The`dashboard_template` folder contains an example monitoring dashboard template that combines standard AI Platform Prediction metrics with log-based metrics defined in the previous steps. You can use [Python Client for Cloud Monitoring Dashboards API](https://googleapis.dev/python/monitoring-dashboards/latest/index.html) to create a dashboard based on the template.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Load the dashboard template"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dashboard_service_client = DashboardsServiceClient(credentials=creds)\n",
    "parent = 'projects/{}'.format(PROJECT_ID)\n",
    "\n",
    "dashboard_template_file = 'monitoring-template.json'\n",
    "with open(dashboard_template_file) as f:\n",
    "    dashboard_template = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Create the dashboard in Cloud Monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dashboard_proto = Dashboard()\n",
    "request = {\n",
    "    'parent': parent,\n",
    "    'dashboard': dashboard_proto,\n",
    "}\n",
    "dashboard_proto = ParseDict(dashboard_template, dashboard_proto._pb)\n",
    "dashboard = dashboard_service_client.create_dashboard(request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for dashboard in dashboard_service_client.list_dashboards({'parent': parent}):\n",
    "    print('Dashboard name: {}, Dashboard ID: {}'.format(dashboard.display_name, dashboard.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Deploying Locust to a GKE cluster\n",
    "\n",
    "Before proceeding, you need access to a GKE cluster. You can find a command to create a GKE cluster in [Environment setup](https://github.com/GoogleCloudPlatform/mlops-on-gcp/blob/master/model_serving/caip-load-testing/README.md#environment-setup) section of [README.md](https://github.com/GoogleCloudPlatform/mlops-on-gcp/blob/master/model_serving/caip-load-testing/README.md). The described deployment process can deploy Locust to any GKE cluster as long as there are enough compute resources to support your Locust configuration. The default configuration follows the Locust's best practices and requests one processor core and 4Gi of memory for the Locust master and one processor core and 2Gi of memory for each Locust worker. As you run your tests, it is important to monitor the the master and the workers for resource utilization and fine tune the allocated resources as required.\n",
    "\n",
    "The deployment process has been streamlined using [Kustomize](https://kustomize.io/). As described in the following steps, you can fine tune the baseline configuration by modifying the default `kustomization.yaml` and `patch.yaml` files in the `locust/manifests` folder.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Install Kustomize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -s \"https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh\"  | bash\n",
    "!sudo mv kustomize /usr/local/bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Set credentials to access your GKE cluster\n",
    "\n",
    "Use, the `gcloud` command to set credentials to your GKE cluster. Make sure to update the `cluster_name` and `cluster_zone` variables with values reflecting your environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud container clusters get-credentials {GKE_CLUSTER_NAME} --zone {GKE_CLUSTER_ZONE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Build the Locust image\n",
    "\n",
    "The first step is to build a docker image that will be used to deploy Locust master and worker pods. The image is derived from the [baseline locust.io image](https://hub.docker.com/r/locustio/locust) and embeds the locustfile and the files's dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_uri = 'gcr.io/{}/locust'.format(PROJECT_ID)\n",
    "\n",
    "!gcloud builds submit --tag {image_uri} locust/locust-image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4. Update the manifests\n",
    "\n",
    "Before proceeding with deployment, you need to update the default manifests. The manifests are located in the `locust/manifests` folder. You will modify two files: `kustomization.yaml` and `patch.yaml`.\n",
    "\n",
    "1. **Set the name of the custom Locust image** - Update the `kustomization.yaml` file with a reference to the custom image you created in the previous step.  Update the `newName` field in the `images` section of the `kustomization.yaml` file.\n",
    "\n",
    "2. **Set the number of worker pods** - The default configuration deploys 32 worker pods. If you want to change it, modify the `count` field in the `replicas` section of the `kustomization.yaml` file. Make sure that your GKE cluster has as sufficient CPUs for the Locust workers.\n",
    "\n",
    "3. **Set the GCS bucket for the test configuration and data files** - When you start a test, the locustfile retrieves a test configuration and test data files from a GCS location. Modify the `configMapGenerator` section of the `kustomization.yaml` file to set the `LOCUST_TEST_BUCKET`, `LOCUST_TEST_CONFIG`, and `LOCUST_TEST_PAYLOAD` literals to the GCS bucket name, the test config file name, and the test data config file name respectively.\n",
    "\n",
    "4.  **Modify the node pool that hosts the Locust master and workers** - Update the name of the node pool in the `patch.yaml` file to which the Locust workload will be deployed. By default, the node pool name is `default-pool`. The name of the node pool is a value of the `values` field in the `matchExpressions` section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat locust/manifests/kustomization.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat locust/manifests/patch.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5. Deploy Locust\n",
    "\n",
    "You are now ready to deploy Locust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kustomize build locust/manifests |kubectl apply -f -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Configure a Locust test\n",
    "\n",
    "At the start of each test, the locustfile script attempts to retrieve test data and a test configuration from a GCS location. Both the test data and the test configuration are formated as JSON. \n",
    "\n",
    "The test data is an array of JSON objects, where each object includes a list of instances and a model signature. If the array contains more than one object, Locust users will randomly pick a list of instances and an associated signature with each call to the `predict` method of the AI Platform Prediction endpoint.\n",
    "\n",
    "The test configuration is a JSON object with a project id, model name, model version, and a test id."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Prepare test data\n",
    "\n",
    "In this example we are using the  **ResNet101** model prepared and deployed in the `01-prepare-and-deploy.ipynb` notebook. We will prepare the instances for the `serving_preprocess` signature of the model using a couple of JPEG images from the `test_images` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = []\n",
    "for image_name in os.listdir(IMAGES_FOLDER):\n",
    "    with open(os.path.join(IMAGES_FOLDER, image_name), 'rb') as f:\n",
    "        images.append(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "single_instance = [{'b64': base64.b64encode(images[0]).decode('utf-8')}]\n",
    "two_instances = [{'b64': base64.b64encode(image).decode('utf-8')} for image in images] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = [\n",
    "        {\n",
    "            'signature': 'serving_preprocess',\n",
    "            'instances': single_instance\n",
    "        },\n",
    "        {\n",
    "            'signature': 'serving_preprocess',\n",
    "            'instances': two_instances\n",
    "        }\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data_local_file = 'test-payload.json'\n",
    "\n",
    "with open (test_data_local_file, 'w') as f:\n",
    "    json.dump(test_data, f)\n",
    "    \n",
    "!gsutil cp {test_data_local_file} {GCS_LOCUST_TEST_CONFIG_DIR}/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Prepare test config\n",
    "\n",
    "Make sure to update the below mapping with the values representing your environment. The `test_id` is an arbitrary value that is used to match the custom log-based metrics records with a given test run. Use a different value anytime you start a test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_id = 'test-{}'.format(datetime.datetime.now().strftime('%Y%m%d-%H%M%S'))\n",
    "\n",
    "test_config = {\n",
    "    'test_id': test_id,\n",
    "    'project_id': PROJECT_ID,\n",
    "    'model': MODEL_NAME,\n",
    "    'version': MODEL_VERSION\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_config_local_file = 'test-config.json'\n",
    "\n",
    "with open (test_config_local_file, 'w') as f:\n",
    "    json.dump(test_config, f)\n",
    "\n",
    "!gsutil cp {test_config_local_file} {GCS_LOCUST_TEST_CONFIG_DIR}/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gsutil ls {GCS_LOCUST_TEST_CONFIG_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Run the Locust load Tests\n",
    "\n",
    "Load tests can be configured, started, monitored and stopped using using Locust's [web interface](https://docs.locust.io/en/stable/quickstart.html#locust-s-web-interface). \n",
    "\n",
    "In our deployment, the web interface is exposed by an external load balancer. You can access the interface using the following URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "IP_ADDRESS=$(kubectl get service locust-master | awk -v  col=4 'FNR==2{print $col}')\n",
    "echo http://$IP_ADDRESS:8089"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can try using the following parameter configurations:\n",
    "1. Number of total users to simulate: 152\n",
    "2. Spawn rate: 1\n",
    "3. Host: `http://[your-region]-ml.googleapis.com`\n",
    "\n",
    "**NOTE**: `[your-region]` is the region for deploying the model that you configured as `REGION` in the first notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up: remove the Locust deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kustomize build locust/manifests | kubectl delete -f -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next step\n",
    "\n",
    "After a test completes, the test's metrics are retrieved from Cloud Monitoring and consolidated into a Pandas dataframe to facilitate comprehensive post-mortem analysis, using the `03-analyze-test.ipynb` Notebook. Please review the [Data Retention](https://cloud.google.com/monitoring/quotas#data_retention_policy) policies of Cloud Monitoring.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License."
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-2-2-gpu.2-2.m50",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-2-2-gpu.2-2:m50"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
